Preliminary Evaluation of the Voyager Spoken Language System

VOYAGER is a speech understanding system currently under development at MIT. It provides information and navigational assistance for a geographical area within the city of Cambridge, Massachusetts. Recently, we have completed the initial implementation of the system. This paper describes the preliminary evaluation of VOYAGEi% using a spontaneous speech database that was also recently collected. I N T R O D U C T I O N One of the important factors that have contributed to the steady progress of the Strategic Computing Speech program has been the establishment of standardized performance evaluation procedures [1]. With the use of common databases and metrics, we have been able to objectively assess the relative merits of different approaches and systems. These practices have also had a positive influence on the natural language community in that databases and rigorous evaluation procedures for natural language systems are beginning to emerge. As we move towards combining speech recognition and natural language technology to achieve speech understanding, it is essential that the issue of performance evaluation again be addressed early on, so that progress can be monitored and documented. Since the Spoken Language Systems program is in its infancy, we do not as yet have a clear idea of how spoken langauge systems should be evaluated. Naturally, we should be able to benefit from hands-on experience with applying some candidate performance measures to working systems. The purpose of this paper is to document our experience with the preliminary evaluation of the VOYAGEa system currently under development at MIT, so that we may contribute to the evolutionary process of defining the appropriate evaluation measures. VOYAGER is a speech understanding system that can provide information and navigational assistance for a geographical area within the city of Cambridge, Massachusetts. The components of the system are described in a companion paper [2]. To evaluate VOYAGER we made use of a spontaneous speech database that we have recently collected consisting of nearly 10,000 sentences from 100 speakers. The database is described in another companion paper [3].